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What I’ve learned, So Far, about Evaluation 



Introduction 

I have been evaluating programs for some 35 years now, and I still enjoy doing that, but I 
have acquired a little insight regarding the effort. I would like to share 5 things that I have learned, 
so far. These are: 

1 . Make sure that the Evaluator is involved early on. 

2. The Evaluator and the Program Director should work jointly on the evaluation. 

3 . Evaluators need to combine hard and soft data. 

4. The Evaluator must learn to compromise. 

5. Most Program Directors do not want to be evaluated. 

1. Make sure that the Evaluator is involved early on. 

It is very crucial that the Evaluator is involved early on in the planning of the program so 
that the evaluation will dovetail with the program. The many evaluations that I have been asked 
to do when the program is almost finished are very frustrating and yield little information. The 
data collected is often not very relevant to the objectives, and is often fragmentary. The 
evaluation is usually one that is required and the Program Director is not interested in the results, 
only in getting a report to the funding agency. Needless to say, these last minute evaluations are 
usually one-year contracts and the relationship is not continued in future years. In most cases, 
the program has been funded for only one year as well. 

There are three techniques that I have used to get myself on board early on. First, 
involvement in multiple year programs usually facilitates early involvement, especially after year 
1 . The five-year plan in Appendix A is a good way to provide the Program Director with the 
picture of what the evaluation will entail over those five years. This plan was, in fact, modified 
somewhat from year-to-year, but at least both the Evaluator and the Program Director had a 
pretty good picture of where the evaluation was headed. 

Second, a planning meeting with the Program Director before the program starts in the 
next cycle (usually year) can focus on what has been changed since the last cycle. Maybe a new 
component has been included. Or more time is planned for one activity. Whatever the change is, 
if the Evaluator can get the Program Director to think about determining if that change produces a 

better program, then one purpose for the evaluation has been determined. And that purpose has 
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been jointly determined by the Evaluator and the Program Director. In actuality, this process 
works better if several program people meet with the Evaluator. There is less likelihood that the 
program people will feel intimidated, and the program people can provide each other with ideas. 

When I was an internal Evaluator for the Dallas public schools, I evaluated a Title I 
program called A Priori. The program developers were constantly trying to improve the program, 
and in this case it was a very good program. The developers were quality people and so was the 
program. But they always began the first meeting each year with, “Well we have made these two 
changes, and we are particularly interested to see if they are making our program even more 
effective.” What a wonderful way to start out each evaluation cycle— an invitation to evaluate the 
program, as well as identification of the focus for that cycle! 

2. The Evaluator and the Program Director should work jointly on the evaluation. 

Evaluators should have technical skills that Program Directors would not be expected to 
have. But Evaluators can be more effective if they work in consort with Program Directors. 
Evaluators do not know the program as well as the Program Director, nor do they know the 
Stakeholder as well. In many cases there are constraints on the Program Director or on the 
program that the Evaluator is unaware of. The days of the Evaluator riding in on the white horse 
and identifying and communicating truth are over (those days really never existed, though we 
may have pretended they did). One reason that I think evaluations have seldom been attended to 
is that a “technical expert” was trying to tell the Program Director how to run the program. 
Isadore Newman and I have written a draft textbook based on the premise that the Evaluator and 
the Program Director need to work together on almost every evaluation task. Indeed, the 
Evaluator usually takes the lead on most tasks, but does it make sense that the Evaluator would 
be the only one to report back the evaluation results to each of the Stakeholder groups? Or that 
the Program Director would not be involved in determining the scope of the evaluation plan? 
Appendix B contains our very approximate estimate of the percentage effort of each person on 
each task. 

Several examples of reporting results might be informative. I had written the draft of the 
final report that summarized information across the 30 Head Start sites. In reviewing the draft 
report, the Program Director indicated that she was also interested in the results by site. As an 
outsider, I did not have that interest, but clearly she did. It took very little effort to analyze and 
report the results by site. 
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Another example of working hand-in-hand with the Program Director is attending 
Advisory Board meetings. My experience is that I can gain much information about the program 
and the pressures and constraints on the program by attending Advisory Board meetings. These 
meetings also offer an opportunity to share evaluation results and to gain the political support 
for evaluation. 

I would like to share two examples of involving program staff in the reporting of 
information. Evaluators are notorious for producing voluminous reports that are of interest to 
them, but are often of little interest and value to the Program Director and of absolutely no value 
to the front-line staff who are implementing the program. Appendix C contains a caricature of the 
A Priori classroom that we shared with the A Priori teachers after we had made a round of 
classroom visitations. It should be noted that the A Priori program was a scripted curriculum, 
which the teachers were supposed to follow on a daily schedule. It was a pull-out program, so 
the teachers repeated themselves about five times each day. Typical teacher caricature was used 
to report a set of classroom implementation results. The typical “teacher” was constructed, as in 
Exhibit 12.3. The typical “nerd” or typical “valley girl” provided the inspiration for this 
technique. Process information that the Evaluators wanted to report to teachers was incorporated 
into the drawing of the teacher and surroundings. For instance, the teacher was wearing an “I 
Love A Priori” sweatshirt showing that the teachers were very enthusiastic about the program 
called “A Priori.” On the other hand, the trash basket had become the resting place for one poem 
that was supposed to be used during the observational period, but was not used by any teachers. 
Teachers participated in discovering the messages in the typical “teacher.” The picture was 
crudely drawn, adding a further touch of informality to the method. 

Teachers were handed the drawing and asked to figure out what it depicted. Teachers 
enthusiastically tried to discover the hidden messages. Teachers easily grasped the notion that 
each was not being depicted, that the picture was a summary of all the teachers that the 
Evaluators had seen. Although only a small part of the evaluation information was shared in this 
way, the teachers were provided with a visual image of the results that was easy to interpret, and 
that communicated evaluation information that was of relevance to them. Since the goal of any 
reporting effort should be to maximize the amount of information that the Stakeholders retain 
(rather than maximize the amount of information that the Stakeholders are given), this reporting 
method seems to have face validity as a useful method. One can only guess how many teachers 
pinned the picture on their bulletin board. 

3 




5 



The Gong Show was an unabashed attempt for the developer of many game shows, 

Chuck Barris, to have fun by doing his own thing on his own TV Show. Chuck Barris was the 
host, and contestants were allowed to show their talents on national TV. A panel of celebrities 
judged the talent on a 1 to 10 scale. When the talent was judged bad enough, the contestant was 
gonged and not allowed to finish the act. The gonger had to defend the reason for gonging, which 
usually did not take much explanation, and made for rich humor. 

With my impetus, a team of three Evaluators and three Program Directors planned a Gong 
Show presentation for reporting classroom observations of the implementation of several 
teaching strategies. The Program Directors had previously trained the teachers on the specific 
teaching techniques. The Evaluators were responsible for determining if and how well the 
teachers were currently implementing the strategies in the classrooms. Essential to successful 
implementation was the correct order of components in each strategy, as well as the correct 
implementation. 

Each Program Director began to implement one strategy correctly and then purposefully 
either did something wrong, used components out of order, slowly self-destructed, or 
disintegrated. One other Program Director would gong the act and explain why the act was 
gonged. Then one Evaluator shared whether these same problems were or were not observed in 
the classrooms. 

Although this reporting method required extensive planning between the Evaluators and 
the Program Directors, the teachers enthusiastically received it. In reality, the functions of staff 
development and evaluation had been blended. The teachers received a different dose of the 
strategies. The teachers became involved with the Gong Show, watching closely for mistakes, 
even though they respected the Program Directors very much. Teachers began to hiss when 
mistakes occurred, and the hissing got louder when the Program Director was way off the 
accepted implementation process. 

During the planning of the Gong Show, one of the Program Directors was somewhat 
hesitant to use this approach with the group for which she was responsible. However, when she 
saw the reaction of the teachers and realized the staff development benefits of this reporting 
method, she requested that her group also be given the Gong Show. 

While those who conducted the Gong Show were positive about the experience, such a 

reporting method would not always work. For instance, there was a lot of planning required 

between two units in an organization that often did not communicate well. In addition, such a 
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presentation requires a certain personality of the presenters, and certain camaraderie between the 
audience and the presenters. Finally, the impact of the Gong Show method would seriously be 
diminished if the content of the report was not appropriate or if the method had recently been 
used. For the right time and the right place, the Program Directors, the Evaluators, and the 
teachers had some fun, and the teachers improved in their ability to implement the program. 

3. Evaluators need to combine hard and soft data. 

I started evaluating programs in the late 60s. Back in the Dark Ages evaluations was 
viewed as a go-no go endeavor. We did have stables of white horses then and Program Directors 
had every right to fear us. We didn’t have many tools, relying on statistics and research design to 
get us through the evaluation. For me the CIPP model was an eye-opener and I quickly embraced 
the notion of looking at the Context within which the program operated, the Input of funds and 
staff , and Process of implementing the program, as well as the traditional Product of all these 
previous endeavors. While I still feel that the ultimate goal of any program should be to produce 
some result, I strive to be cognizant of the constraints and challenges along the way. The hard 
information must be combined with the soft information. Neither can stand alone. Squeezing out 
impressive results at the expense of the feelings of staff and participants is not good practice. 
Neither is getting bad results from staff and recipients that feel the program is the best thing since 
sliced bread. Appendix D is a compendium of kinds of information that could be obtained from 
various sources. Note that much of the information would be categorized in the ‘soft’ range. 

4. The Evaluator must learn to compromise. 

The hardest lesson that I have learned is to compromise. The compromises are not only a 
function of the available funds, but also the interests of the Program Director. In many cases the 
statistical and research design notions are either not appreciated or are beyond the training of the 
Program Director or the Stakeholder. Evaluators have to speak the language and be willing to 
modify plans. The compromise may be to test fewer participants than desired. In one case I have 
had to try to convince the Program Director to test fewer participants because of the cost and 
time involved in her staff testing the participants. 

5. Most Program Directors do not want to be evaluated. 

I suppose that I didn’t need to identify this fifth point. It’s pretty easy to figure that out 
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early in your career. But I keep hoping that the next evaluation will be like the two that really 
went well as a result of the Program Director wanting to know how well the program succeeded 
and how to make it better along the way. I often try to convince myself that this particular 
evaluation will be “the one.” I try to fight off reality as long as I can. I try to disbelieve reality, 
and continue to hope for the ideal. Appendix E contains a figure depicting what I think is the 
breakdown of Program Directors with respect to how they view evaluation. The positive 
sections are probably not as large in reality. 

Appendix F contains a little scenario regarding how the powers to be tried to intimidate 
me in an evaluation. The outside contractor, who did not want to know the answer to the 
question, ’’Does the million dollars a year spent on this program benefit the children?” still has 
that contract with the Dallas Schools. 

Appendix G depicts the story of how the entire evaluation team got kicked out of staff 
development because we dared to recommend some changes. Those very changes were 
implemented the following staff development, but the evaluation team was not given credit. In 
fact the whole team was reassigned to other programs the following year. 

Next Steps 

Will I continue to evaluate programs? Yes, for five reasons. First, I feel that I often 
represent the taxpayer. The taxpayers are footing the bill, and they need to be reassured that the 
money is being used in the best possible way. Second, I enjoy leaving the Ivory Tower to assist 
State and Federally funded entities. I feel like I am not only contributing my taxes, but making 
sure they are doing the most good. Third, the evaluation contracts keep me in touch with good 
people, both on the program side as well as on the evaluation side. Fourth, the extra dollars come 
in handy. 
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Appendix A 
Five year plan 

Timeline for Evaluation Activities 

Year 1 (focus is on Context and Input) months 

1 . Clarify evaluation questions with Stakeholders 1 

2. Catalog existing data 2 

3. Finalize evaluation plan and get Board approval 1 

4. Design procedures for new data collection 1 

5. Collect new data 2 

7. Develop reports 1 

8. Deliver reports 1 

9. Clarify evaluation for year 2 .2 

Year 2 (focus is on Process) months 

1 . Finalize evaluation plan and get Board approval .3 

2. Catalog existing data .3 

3. Design procedures for new data collection .5 

4. Collect new data 

attend Transition activities 1 

interview LEA staff .5 

interview Board .2 

interview Educational Specialist for Transition .3 

survey parents 1 

interview students 1 

survey LEA 1 

survey Head Start staff .5 

interview Head Start central office staff .2 

5. Analyze all data 3 

6. Develop reports 1 

7. Deliver reports 1 

8. Clarify evaluation for year 3 .2 

Year 3 (focus is on Process) months 

1 . Finalize evaluation plan and get Board approval .3 

3. Design procedures for new data collection .5 

4. Collect new data 

attend Transition activities 1 

interview LEA staff .5 

interview Board .2 

interview Educational Specialist for Transition .3 

survey parents 1 
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interview students 1 

survey LEA 1 

survey Head Start staff .5 

interview Head Start central office staff .2 

5. Analyze all data 3 

6. Develop reports 1 

7. Deliver reports 1 

8. Clarity evaluation for year 4 .2 

Year 4 (focus is on Product) months 

1 . Catalog existing data 2 

2. Finalize evaluation plan and get Board approval . 1 

3. Design procedures for new data collection .8 

4. Collect new data 3 

attend Transition activities 
interview Board 



interview Educational Specialist for Transition 
interview parents 
survey parents 
survey LEA staff 

collect LEA student-level data from records 

5. Analyze all data 3 

6. Develop year 4 report 2.5 

7. Deliver reports .5 



Year 5 (focus is on Product) months 

1 . Catalog existing data 2 

2. Finalize evaluation plan and get Board approval . 1 

3. Design procedures for new data collection .8 

4. Collect new data 3 

attend Transition activities 
interview Board 



interview Educational Specialist for Transition 
interview parents 
survey parents 
survey LEA staff 

collect LEA student level data from records 



5. Analyze all data 3 

6. Develop year 5 report 1 

7. Develop final report 1.5 

8. Deliver reports .5 
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Appendix B 

Approximate involvements of Program Director and Evaluator in each of the 39 tasks of 

the General Evaluation Model 

Task Percent time on task 

Program Director Evaluator 



1 . Identify Stakeholders 


90 


10 


2. Identify program areas 


90 


10 


3. Identify sources of information 


20 


80 


4. Develop the Needs Assessment instrument 


20 


80 


5. Conduct the Needs Assessment 


50 


50 


6. Write the Needs Assessment report 


20 


80 


7. Disseminate to Stakeholders 


80 


20 


8. Make sure Stakeholders buy into program 


90 


10 


9. Determine instrument(s) 


10 


90 


10. Determine comparison group(s) 


10 


90 


1 1 . Administer Baseline 


20 


80 


12. Analyze information 


00 


100 


13. Write report 


20 


80 


14. Disseminate the Baseline report 


30 


70 


15. Draft program objectives 


70 


30 


16. Share program objectives with Stakeholders 


70 


30 


17. Finalize program objectives 


50 


50 


18. Develop procedures 


90 


10 


19. Train staff 


90 


10 


20. Determine that training was implemented well 


20 


80 


21. Develop Program Implementation Evaluation plan 


20 


80 


22. Identify instruments 


20 


80 


23. Develop instruments (if necessary) 


20 


80 


24. Inform Stakeholders periodically 


60 


40 


25. Context information collected and analyzed 


20 


80 


26. Input information collected and analyzed 


40 


60 


27. Process Implementation Assessment 






information collected and analyzed 


30 


70 


28. Write report 


20 


80 


29. Relate to objectives and procedures to achieve 


40 


60 


30. Disseminate the reports 


80 


20 


31. Develop Post Assessment plan 


10 


90 


32. Identify instruments 


10 


90 


33. Develop instruments (if necessary) 


10 


90 


34. Collect Post Assessment information 


05 


95 
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35. Analyze all information 


00 


100 


36. Write End-Of-Cycle report 


20 


80 


37. Relate to objectives and procedures to achieve 


20 


80 


38. Disseminate the report(s) 


60 


40 


39. Determine evaluation for next cycle 


20 


80 
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Appendix C 

Caricature of the A Priori classroom 
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Appendix D 

Compendium of kinds of information that could be obtained from various sources 



Sources of information 



Recipients 



Staff 



Relatives of recipients 



Community members 



Existing records 



Unobtrusive information 



0 




Examples of information 



Performance on standardized tests 
Attitudes 
Career choices 
Attendance at special events 
Attendance at in-service 
In-service implementation 
Attitude about program 
Turnover rate and reasons 
Innovative implementation of program 
Attitudes of recipients 
Behavior outside of program 
Comparison to older siblings 
Perceptions of community 
Attitude toward program goals 
Perceptions of success of program 
Willingness to volunteer in program 
Attendance by recipients and relatives 
Discipline referral rate 
Staff performance evaluations 
Staff turnover 

All above information in comparison locations 

Amount of litter surrounding building 

Respect for “Do Not Walk” on grass signs 

Appropriate noise level (low during quite time, but high during 

get noisy time) 
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Appendix E 

The breakdown of Program Directors with respect to how they view evaluation. 




Appendix F 

Scenario regarding how the powers to be tried to intimidate me in an evaluation 

When I was an inside Evaluator for the DISD, I was assigned to evaluate all aspects of 
several new schools. After a less than positive implementation report was disseminated, I was 
called in to the Superintendent’s office and was told that this program was the most effective 
program that the Superintendent had ever been associated with. An outside contractor who knew 
how to pull political weight was running the program for approximately 1 million dollars a year. 
The Superintendent and I met that first and last time. Both are now gone from the DISD, but the 
political pull contractor is still in business there. 
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Appendix G 



How the entire evaluation team got kicked out of staff development 

I was in charge of evaluating the Dallas Independent School District Chapter 1 program. 
Staff development was held once a month all day on Friday. The topic for several months was 
adding a bilingual component to the existing program. Trainers were hired from the outside. The 
sessions were boring and provided no "hands-on" materials for the participants. Examples for the 
classroom came from teaching French-speaking children in English-speaking schools in Canada 
and the indigenous Mauri in Australia, and not from teaching recent immigrants from Mexico, 
which comprised approximately 90% of the students of the participants. Participants would sign 
in when they arrived in the morning, but many of the 300 were leaving at noon to go back to their 
class (instead of returning for the afternoon continuation). 

The above problems were identified by me to the Program Director, along with 
recommendations to make the sessions more interesting, to provide examples relevant to the 
Mexican population, and a mechanism for keeping attendance in the afternoon. The Program 
Director was upset with the recommendation and the evaluation team was told to not return for 
the remaining staff development sessions. But one of the Evaluators was new to the district, and 
therefore unknown. She was asked by the head Evaluator to return to the next staff development 
session and to keep a low profile among the 300 participants. She reported that (1) attendance 
was taken at both morning and afternoon, (2) door prizes were given based o afternoon 
attendance, (3) and most importantly, the only examples mentioned were related to the Mexican 
children. The evaluation of staff development, particularly checking on assumptions had a 
positive impact on this series of staff development. (Note: the Program Director never did make 
amends and never did share with the evaluation team that these changes were made.) 
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